Add support for missing tasks in mtgp #2960

sdaulton · 2025-08-08T17:57:29Z

Summary:
Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data.

This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair.

This will still error out in Ax if data for the target trial is missing.

Differential Revision: D79812024

facebook-github-bot · 2025-08-08T17:57:38Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

codecov · 2025-08-08T18:08:36Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@9f002eb). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff            @@
##             main     #2960   +/-   ##
========================================
  Coverage        ?   100.00%           
========================================
  Files           ?       216           
  Lines           ?     20321           
  Branches        ?         0           
========================================
  Hits            ?     20321           
  Misses          ?         0           
  Partials        ?         0

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Summary: X-link: pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-08T18:53:56Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Pull Request resolved: pytorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: Pull Request resolved: facebook#4121 X-link: pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-08T22:55:02Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-08T23:05:29Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-09T06:01:35Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-12T06:34:21Z

This pull request was exported from Phabricator. Differential Revision: D79812024

facebook-github-bot · 2025-08-12T06:37:38Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Pull Request resolved: pytorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-12T07:01:00Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-15T18:44:48Z

This pull request was exported from Phabricator. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: facebook/Ax#4121 Pull Request resolved: pytorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 Pull Request resolved: facebook#4121 X-link: pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: pytorch/botorch#2960 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

Summary: X-link: facebookexternal/botorch_fb#24 X-link: facebook/Ax#4121 Currently, cross-validation in Ax fails when using a MTGP if there are multiple metrics and only some metrics have been observed for some tasks. This is a modeling problem, since the model is a ModelListGP and not all MTGPs in the list are required to have the same tasks. Hence when you pass in a test input, the model errors out if there are not observations from that task in the training data. This avoids the error by mapping (optionally) mapping unexpected tasks to the "target task". This does not change the default behavior. For cross-validation in Ax, predictions are discarded if there are no observations for a given (task, metric) pair. This will still error out in Ax if data for the target trial is missing. Differential Revision: D79812024

facebook-github-bot · 2025-08-18T09:09:31Z

This pull request was exported from Phabricator. Differential Revision: D79812024

meta-cla bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Aug 8, 2025

facebook-github-bot added the fb-exported label Aug 8, 2025

sdaulton mentioned this pull request Aug 8, 2025

Add support for missing tasks in mtgp facebook/Ax#4121

Open

sdaulton force-pushed the export-D79812024 branch from f03dda4 to fae6912 Compare August 8, 2025 18:50

sdaulton force-pushed the export-D79812024 branch from fae6912 to 3ad186d Compare August 8, 2025 18:54

sdaulton force-pushed the export-D79812024 branch from 3ad186d to 5ef3c77 Compare August 8, 2025 22:54

sdaulton force-pushed the export-D79812024 branch from 5ef3c77 to 023b232 Compare August 8, 2025 23:05

sdaulton force-pushed the export-D79812024 branch from 023b232 to 6c977ee Compare August 9, 2025 06:01

sdaulton force-pushed the export-D79812024 branch from 6c977ee to fc0e7fd Compare August 12, 2025 06:33

sdaulton force-pushed the export-D79812024 branch from fc0e7fd to 09dd4dd Compare August 12, 2025 06:34

sdaulton force-pushed the export-D79812024 branch from 09dd4dd to bd43641 Compare August 12, 2025 06:37

sdaulton force-pushed the export-D79812024 branch from bd43641 to aee2ced Compare August 12, 2025 07:00

sdaulton force-pushed the export-D79812024 branch from aee2ced to 589ff95 Compare August 15, 2025 18:40

sdaulton force-pushed the export-D79812024 branch from 589ff95 to b51fc3c Compare August 15, 2025 18:44

sdaulton force-pushed the export-D79812024 branch from b51fc3c to 2470005 Compare August 18, 2025 09:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for missing tasks in mtgp #2960

Add support for missing tasks in mtgp #2960

Uh oh!

sdaulton commented Aug 8, 2025

Uh oh!

facebook-github-bot commented Aug 8, 2025

Uh oh!

codecov bot commented Aug 8, 2025 •

edited

Loading

Uh oh!

facebook-github-bot commented Aug 8, 2025

Uh oh!

facebook-github-bot commented Aug 8, 2025

Uh oh!

facebook-github-bot commented Aug 8, 2025

Uh oh!

facebook-github-bot commented Aug 9, 2025

Uh oh!

facebook-github-bot commented Aug 12, 2025

Uh oh!

facebook-github-bot commented Aug 12, 2025

Uh oh!

facebook-github-bot commented Aug 12, 2025

Uh oh!

facebook-github-bot commented Aug 15, 2025

Uh oh!

facebook-github-bot commented Aug 18, 2025

Uh oh!

Uh oh!

Add support for missing tasks in mtgp #2960

Are you sure you want to change the base?

Add support for missing tasks in mtgp #2960

Uh oh!

Conversation

sdaulton commented Aug 8, 2025

Uh oh!

facebook-github-bot commented Aug 8, 2025

Uh oh!

codecov bot commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

facebook-github-bot commented Aug 8, 2025

Uh oh!

facebook-github-bot commented Aug 8, 2025

Uh oh!

facebook-github-bot commented Aug 8, 2025

Uh oh!

facebook-github-bot commented Aug 9, 2025

Uh oh!

facebook-github-bot commented Aug 12, 2025

Uh oh!

facebook-github-bot commented Aug 12, 2025

Uh oh!

facebook-github-bot commented Aug 12, 2025

Uh oh!

facebook-github-bot commented Aug 15, 2025

Uh oh!

facebook-github-bot commented Aug 18, 2025

Uh oh!

Uh oh!

codecov bot commented Aug 8, 2025 •

edited

Loading